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ABSTRACT 

This paper presents the causal comparative method, or 
ex post facto research design, as an alternative to classical 
experimental methods for establishing causal relationships between 
events and circumstances. A literature survey, conducted in an effort 
to define and describe the method is discussed. Following this is a 
presentation of the relationship between the correlation method and 
the causal comparative methods Both the strengths and weaknesses of 
the ex post facto research designs are detailed. (WDR) 
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ABSTRACT 



Employing the experimental method in research is 
sometimes impractical or prohibitively costly in time, 
money, and effort; in other instances, it is unethical 
or immoral. The causal comparative method or ex post 
facto research design, outlined in this paper, is an 
alternative method for establishing causal relation- 
ships between events and circumstances. The literature 
was sxirveyed in an effort to define and describe the 
method. The relationship between the correlation 
method and the causal comparative method is also dis- 
cussed. Both the strengths and the weaknesses of 
ex post facto research designs are detailed. 
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SPECIAL REPORT No. 7320 
COMPUTER-BASED PROJECT for the EVALUATION 
of MEDIA for the HANDICAPPED 

T i t I G I "EXPOST FACTO STUDIES AS A RESEARCH METHOD'* 

BY: Harold G. Lord 
BACK G ROUND 

The Computer Based Project for the Evaluation of Media for the Handicapped, 
based on contract #OEC-9-423617-4357 (616) between the Syracuse (N.Y.) City School 
District and the Media Services and Captioned Films Branch, Bureau of Education 
for the Handicapped (United states Office of Education) for the five year period 
July 1, 1969 through June 30, 1974. The major goal is to improve the instruction 
of handicapped children through the development and use of an evaluation system 
to measure the instructional effectiveness of films and other materials with 
educeble mentally handicapped (EMH) children, in-service training and media support 
for special teachers, and studies related to the evaluation process and the 
populations used. 



The Project has concentrated on the 600 films and 200 filmstrips from the 
Media Services and Captioned Films (BEH - USOE) depository; however, specific 
packages from Project LIFE, various elementary math curricula, and selected 
programs from Children's TV Workshop have* also been evaluated. The evaluation 
model used requires that: 1) objectives of materials be specified and written/ 

2) instruments bo constructed to test and measure effectiveness; andr 3) children 
be the major sources of evaluation information. A number of instruments and 
methodologies are employed in the gathering of cognitive and affective data from 
000 EMH children and 80 special teachers to make the effectiveness decisions. 
Over half of the EMH population can neither read or write; therefore, a unique 
Student Response System (SRS) in employed, consisting of a twenty station G.E.- 
1000 SRS which can be operated in a group or individual recording node and is 
connected to a remote computer system* The computer capabilities consist of 
remote telephone connections to the Rome (N.Y.) Air Development Command, the 
Honeywell time-shared network, and the Schenectady (N.Y.) G E Research and 
Development Center; and batch mode capabilities of the Syracuse City Schools, 
Syracuse University, and various commercial sources. 

In-service and media support activities provide on-the-job training for 
teachers, teacher aides, equipment, and materials to the special teachers in 
the city schools. The research activities have centered around investigations 
and special problems related to the development of the evaluation model. The 
four major areas considered are: 1) testing effects, 2) captioning effects, 

3) special student characteristics; and, 4) evaluation procedures validation. 

Documentation of the major activities appear in the five annual reports 
and the 600 evaluations prepared on materials used. Staff members v;ere encouraged 
to prepare special reports and the attached paper is one of these. The opinions 
expressed in tiiis publication do not necessarily reflect the position or policy 
of the computer Based Project, the United States Office of Education, or the 
Syracuse City School District, and no official endorsement by any of the agencies 
should be inferred. 
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Experimental educational research has been derived 
from the laboratory method often used in the natural sciences 
According to Galfo and Miller (1970) in its most elementary 
form, the experimental method of science is based upon two 
assumptions regarding variables v;hich may be identified with 
the phenomenon under investigation: 

1. If two situations are equal in every respect 
except for a factor present in one of the sit- 
uations, any difference which appears between 
the tv/o situations can be attributed to the 
factor. This statement is referred to as the 
law of the single variable . 

2. If two situations are not equal but it can be 
demonstrated that none of the variables are 
significant in producing the phenomenon under 
investigation; or if significant variables are 
made equal, any difference occurring between 
the two situations after the introduction of 

a new variable to one of the systems can be 

attributed to the new variable. This statement 

is referred to as the law of the only signif i -- 

cant variable [p, 17] . 

The purpose of establishing experimental-control conditions 
is to create a situation in which the effect of a single 
variable can be studied, 

in the behavioral sciences many problems cannot be 
solved by the experimental method. Because of the complex- 
ity and nature of social phenomena, one cannot always select 



and control the factors necessary to study cause and effect 
relations in an artificial laboratory situation. If an ex- 
perimenter controls variations in all but a single independent- 
experimental-variable, he may prevent the simultaneous function 
ing of variables that normally are found operating together: 
and thus may free the independent variable from the influence 
of the other variables. In such instances. an analysis of what 
actually does happen in a natural situation may be a more sat- 
isfactory way to study causes. 

Employing the experimental method is sometimes imprac- 
tical or prohibitively costly in time, money, and effort in 
some instances, such as studies of riots or "life as it is 
lived," In other instances employing the experimental method 
is unethical or immoral. One cannot justify staging fatal 
airplane crashes to ascertain their causes, or placing emo- 
tionally stable children in controlled environments to deter- 
mine whether it is possible to produce various psychoses. Re- 
spect for living things thus prevents an investigator from in- 
flicting unnecessary pain, hardship, or harm on others^ or 
from interfering in any way with the normal growth and devel- 
opment of an individual. Hence, in some cases, experiments 
cannot be utilized to study causation; thus, one must turn to 



the causal comparative method or the ex post f acto research 



design, 

causal Comparative Method 

According to Van Dalen (1962) the causal comparative 

method of research provides insights into life situations; 

Instead of arranging a controlled laboratory ex- 
periment and inducing subjects to do or become some- 
thing , a researcher studies a life situation in 
which subjects are playing, experiencing, or being 
what he vrants to investigate. To study school in«- 
tegration riots, for example, rather than setting 
up an experiment to test whether various factors 
will cause a riot, he compares a community that has 
experienced a riot with one that ha;i not. After 
studying the likenesses and differences between the 
two situations, he describes the factors that ap- 
pear to account for the riot in the one instance 
and not in the other (p. 201] . 

A search is made for factors or conditions which seem to be 

associated with one group and not the other that might serve 

as a possible explanation of the underlying causes. 

The causal comparative method or e_x postp facto method 

of research seeks to establish causal relationships between 

events and circumstances. In other words, it finds out the 

cause of certain occurrences or non-occi;rrences , This is 

achieved by comparing the circumstances associated with ob- 



served effects and by noting the factors present in the in- 
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stances where a given effect occurs and where it does not 

occur. Sukhia, iMetrotra, and Metrotra (1966) placed the 

iT^ethod in historical perspective within logic: 

This metliiod is based on Mill's Canon of Agreement 
and Disagroement which states that causes of a given 
observed effect may be ascertained by noting ele- 
ments which are invariable present when the result 
is present and which is invariai:)ly absent when the 
result is absent [p. 215], 

The causal^comparative app.roach differs from the his- 
torical, the normative-survey^ and the experimental in vari- 
ous respects. Historical research may also seek to find out 
the causes of events, but while historical research finds out 
the causes of past events, the causal-comparative deals with 
the present events only. Normative-survey deals with present 
events, and it is mainly, concerned with status and trends, 
not the causes of educational effects. The causal compara- 
tive method differs from the experimental procedure in that 
it does not control the variable factors. It, instead, makes 
observations under normal field conditions and discovers the 
causes of observed phenomena. 
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Definition of Ex Post Facto 



Kerlinger (196^) defined ex post facto research as; 

that research in which the independent variable or 
variables have already occurred and in which the 
researcher starts with the observation of a dependent 
variable or variables. He then studies the inde- 
pendent variables in retrospect for their possible 
relations to, and effects on, the dependent variable 
or var iables (p. 3^0) . 

Ary, Jacobs, and Razavieh (1972) suggested that the basic purpose of 

ex post facto research is to discover or establish causal or functional 

relationships among variables, and that "careful investigators prefer 

to speak of functional rather than causal relations (p. 26^).'' 



Tuckman (1572) defined the term ex post facto to be: 

an experiment in which Ihe researcher examines the 
effects of a natural 1 st ical ly-occurr i ng treatment 
after that treatment has occurred rather than creat- 
ing the treatment itself. The experimenter attempts 
to relate this after-the-fact treatment to an out- 
come or dependent measure. While the naturalistic 
or ex post facto experiment may not always be dia- 
grammed^rorn other designs, it is different in that 
the treatment is included by selection rather than 
manipulation, For this reason> it is not always 
possible to assume a simple causative relation between 
independent and dependent variables. If the relation- 
ship fails to be obtained, then it is likely that no 
causative relationship holds. But if the predicted 
relationship is obtained, this does not necessarily 
mean that the variables studies are causally related 
(pp* 



Elements of the Method 



The sequence of steps involved in ex post facto re-- 
search were set forth by Isaac and Michael (1971) , and they 
include: 

1. Define the problem, 

2. Survey the literature. 

3. State the hypotheses. 

4. List the assumptions upon which the hypotheses 
and procedures will be based. 

5. Doisign the approach: 

a. Select appropriate subjects and source materials. 

b. Select or construct techniques for collecting 
the data. 

c. Establish categories for classifying data that 
are unambiguous, appropriate for the purpose 
of the study, and capable of bringing out sig- 
nificant likenesses or relationships. 

6. Validate the data-gathering techniques. 

7. Describe, analyze, and interpret the findings in 
clear, precise terms [p. 23]. 

Sukhia, Metrotra, and Metrotra (1966) listed three 
important aspects of the causal-comparative method in regard 
to its treatment of datat 

1. gathering of data on factors invariably present 
in cases where the given result occurs and dis- 
carding of those elements which are not univer- 
sally present; 

2. gathering of data on factors invariably present 

in cases where the given effect does not occur; and 

3. comparing the two sets of data, or in effect, 
subtracting one from the other to get at the causes 
responsible for the occurrence or otherwise of the 
effect [p. 2,15] . 
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By their very nature, ex post facto experiments can 
provide support for any number of different and perhaps con- 
tradictory, hypotheses; they are so completely flexible that 
it is largely a matter of postulating hypotheses according to 
one's personal preferences. The point is that the evidence 
simply illustrates a hypothesis; it does not test the hypothe- 
sis since hypotheses cannot be tested on the same data from 
which they were derived. The relationship noted may actually 
exist but it is not necessarily the only relationship, or 
perhaps even the crucial relationship. 

This docs not mean that the ex porjt facto experiment 
is completely worthless; many of our important studies in ed- 
ucation and psychology are ex post facto studies. On the 
other hand, their failure to introduce the common elements 
of control, randomization, etc. makes them extremely vulner- 
able from a scientific point of view and the danger of their 
being misleading should be clearly recognized. Ex post facto 
•^experiments" are probably better conceived as surveys, use- 
ful in the derivation of hypotheses to be tested through more 
conventional experimental approaches. 



Correlation 

Related to the causal comparative method of research 
is correlation. The correlation method of research is not 
merely a statistical device of calculating degrees of rela- 
tionships (coefficients of correlation) between certain data 
It is a research approach^ which analyses the relationship 
between data^ between variables and some results in such a 
way that the underlying pattern of relationships becomes 
clear. 

In their discussion of correlation, Weinberg and 
Schumaker (1969) set forth the following formal definition: 

Two variables are said to correlate positively when, 
as one variable increases, the other shows some 
trend to increase correspondingly in a uniform way. 
Two variables are said not to correlate at all when, 
as one of them changes, the other shows some trend 
to decrease correspondingly in a uniform way [p. 248] 

Correlation refers to the degree to which two variables move 

uniformly with respect to each other. Correlation, since it 

is linear in nature, may be shown graphically as some point 

on a continuum. If a line were drawn to represent possible 

correlations, the relationship between any two variable may 

be shown as some point on that line. 
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Perfect positive correlation 
4 - Some positive correlation 
^ ' ■ ' No correlation 

4 Some negative correlation 

^^^^ Perfect negative correlation 

Weinberg and Schumaker stated this warning: 

Remember that when we discuss correlation we are 
always talking about a trend that exists and not a 
necessary or invariable relationship. Also remember 
that the fact that two variables correlate does not 
at all imply that either variable causes the other 
to change. In any given case, the relationship may 
be coincidental or the consequence of some external 
cause that has not even been discovered [p, 248]. 

Quas-Experimental Des ign 

Mouly, in his book, The science of educational research 

(1970), indicated his belief that tho ex post factd experiment 

is of questionable quas-exper imental design: 

A relatively questionable quasi-experimental 
design is the ex post facto experiment, in which a 
particular characteristic of a given group is in-- 
vestigated with a view to identifying its antece- 
dents. This is experimentation in reverse: instead 
of taking groups that are equivalent and exposing 
them to different treatment with a view to promoting 
differences to be measures, the ej< post fact o experi- 
ment begins with a given effect and seeks the experi- 
mental factor that brought it about. The obvious 
weakness of such an ^'experiment'' is that we have no - 
control over the situations that have already oc- 
: = curred and we can never be sure of how many other 

circumstances might have been involved [p. 340). 



10 



Isaac and Michael (1971) offerred the following strengths 
and weaknesses for the e>c post fa cto research design: ' 
Strengths 

1. The cau3al-(2omparative method is appropriate in 
many circumstances where the more powerful experi- 
mental method is not possible: 

a. When it is not always possible to select, 
control, and manipulate the factors necessary 
to study cause-and-ef feet relations directly. 

b. When the control of all variations except a 
single independent variable may be highly 
unrealistic and artificial, preventing the 
normal interaction with other influential 
variables . 

c. When laboratory controls for many research 
purposes would be impractical, costly, or 
ethically questionable. 

2. It yields useful information concerning the nature 
of phenomena: what goes with what, ^unde?: what con- 
ditions, in what sequences and patterns, ancf *fche 
like, 

3. Improvements in techniques, statistical methods, 
and designs with partial control features, in 
recent years, have made these studies more de- 
fensible. ' 

Weaknesses 

1* The main weakness of any ex post facto design is 
the lack of control over independent variables. 
Within the limits of selection, the investigator 
must take the facts as he finds them with no 
opportunity to arrange the conditions or manip- 
ulate the variables that influenced the facts in 
the first place. To reach sound conclusions, 
the investigator must consider all the other pos- 
sible reasons or p lausible rival hypotheses which 
might account for the results obtained. To the 
' ..r ext<int that he can successfully justify his con- 
clusions against these other alternatives, he is 
in a position of relative ritrength. 



2. The difriculhy in being cerhain that the relevant 
causative factor is actually Included among the 
many factors under study. 

3. The complication that no single factor is the 
cause of an outcome but some combination and 
interaction of factors go together under certain 
conditions to yield a given outcome. 

4. A phenomenon may result not only from multiple 
causes but also from one cause in one instance 
and from another cause in another instance. 

5. When a relationship between two variables is dis- 
covered^ determining which is the cause and which 

* the effect may be difficult. 

6. The fact that two, or more/ factors are related 
does not necessarily imply a cause-and-ef feet re- 
lationship. They all simply may be related to an 
additional factor not recognized or observed. 

7. Classifying subjects into dichotomous groups (e.g., 
"Achievers" and *'Nonachievers") , for the purpose 
of comparison, is fraught with problems, since 
categories like these are vague, variable, and 
transitory. Such investigations often do not 
yield useful findings. 

8. Comparative studies in natural situations do not 
allow controlled selection of subjects. Locating 
existing groups of subjects who are similar in all 
respects except for their exposure to one variable 
is extremely difficult (pp. 22-23]. 

In a philosophical discussion of causation, Broudy, 

Ennis, and Krimerman (1973)* stated: 

causation is a thorny topic for educational research- 
ers, yet one that we must face, and face squarely. 
V^e know how easy it is to commit the post hoc fal- 
facy; to think that just because something followed 
or follows something else, it was or is caused by 
that something else; so we are justifiably cautious 
about making causal allegations. But sometimes • 
this caution is carried to tho extreme: complete 
avoidance of the use of the word ♦'cause" {often solf- 
defeatingly accompanied by use of terms with sirailar 




meaning, like **brings about," or "produces," or 
accompanied by an ambiguous statement of the con- 
clusion so that it is obvious if noncausal, but 
not explicitly causal in nature [p. 401] . 

Even with its weaknesses, the lex post facto research 

design play^; an important role in understanding mankind • 

Broudy, Ennis, and Krimerman (1973) summarized their concern 

for causal relationships: 

V/e cannot avoid concern with causal relationships, 
both general and particular. Education is concerned 
with bringing about changes, with providing general 
guidelines for so doing (guidelines that must take 
the form of causal generalizations) , and with par- 
ticular causal analyses of existing or recent occur- 
rences. Furthermore, a number of educators are in- 
terested in knowing when children have developed a 
concept of causation and/or when they are ready to 
develop this concept. These practical concerns of 
the educator require educational researchers to use 
causal terms. We should be as clear as possible 
about what we are saying when we make a 'causal alle- 
gation and should have a good idea about what counts 
as proof and disproof, and support for and opposition 
to a causal allegation [p. 401]. 

Kelinger (1964) presented a strong case for the research 

method : 

It can even be said that ex post facto research is 
more important than experimental research. This is, 
of course, not a methodological observation. It 
means, rather, that the most important social sci-- 
ontific and educational research problems do not 
lend themselves to experimentation, although many of : 
them do lend themselves to controlled inquiry of the 
ox pos t fac tp kind. Consider the study of the effect 
of the English eleven-plus examination on English 



children; current studies of children*s creativity, 
Piaget^s studies of children's thinking, Gross's 
studies of bo. <s of education and superintendents, 
and the auth rxtarianism studies of Adorno, et^ al , 
If a tally of sound and important studios in psy- 
chology, sociology, and education were made, it is 
likely that ex post facto studies would outnumber 
and outrank experimental studies [p. 373] • 

He later tempered this position by stating that the weaknesses 

of the method must be recognized and considered: 

Some students of research believe that much behavioral 
research, but particularly educational research, suf- 
fers from a serious lack of a rigorous experimental 
approach and that it will lag as long as this situa- 
tion exists. The author believes that good experi- 
mental research is badly needed in all fields, and 
that large doses of poor ex^ post facto research 
should be avoided. Improvements in educational ex 
pos t facto research are badly needed. Perhaps a good 
rule to follow would be to ignore the results of any 
ex post facto study that does not test hypotheses. 
Exceptions to this Gtricture should be few and far 
between. Perhaps another good rule would be to be 
highly skeptical of any ex post facto study that 
tests only one hypothesis; that is, alternative "neg- 
ative" hypotheses should be routinely tested. Re- 
searchers should predict significant relations and 
nonsignificant relations whenever possible . 

A final piece of advice is this: always treat 
the results and interpretations of the data of ex 
pos t facto investigations with great care and cau- 
tion. Where one must be careful with experimental 
results and interpretations, one must be doubly care- 
ful with ex post facto results and interpretations 
[p. 373] , 

Causal comparative studies have many limitations, and 
they often do not produce the precise, reliable knowledge that 



can be gained through good experimental studies. But they do provide 
a means of tackling problems that cannot be probed In laboratory situa- 
tions, and they do yield valuable clues concerning the nature of the 
phenomena, As the techniques, toolSi and controls used In conducting 
causal comparative studies have been improved, this method of inquiry 
has gained greater respect. 
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